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^ . Abstract 

Let /i be a computable ergodic shift-invariant measure over {0, 1}^. Providing 
a constructive proof of Shannon-McMillan-Breiman theorem, V'yugin proved that if 
X e {0, 1}^ is Martin-Lof random w.r.t. fi then the strong effective dimension Dim(x) 
of X equals the entropy of /i. Whether its effective dimension dim(2;) also equals the 
entropy was left as an problem question. In this paper we settle this problem, provid- 
ing a positive answer. A key step in the proof consists in extending recent results on 
tyj I Birkhoff 's ergodic theorem for Martin-Lof random sequences. 
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1 Introduction 



(3 ■ The effective dimension and strong effective dimension of an infinite binary sequence x are 

defined as 



dini(x) = liminf ■ 



n 



H ; T^- ( \ V K{x\n) 

a ^ Dini(x) = lim sup 



n 



where K{w) is the Kolmogorov complexity of w. 

They can be characterized as effective versions of Hausdorff and packing dimensions 
respectively, or by divergence of s-gales (see [LutOO, May02, AHLM07] for the original 
results and [Lut05] for a survey). 

Let p E [0, 1] be a computable real number and Hp the Bernoulli measure over Cantor 
space given by /ip[u7] = pl"'li(l— p)!""!". It is well-known that if an infinite binary sequence x 
is Martin-Lof random w.r.t. /ip then dim(x) = Dim(x) = h(fip), where /i(/ip) is the entropy 
of fip defined by 

hM = -plog(p) - (1 -p)log(l -p). (1) 



This result is not difficult to prove and reduces to the strong law of large numbers for 
Martin-Lof random sequences, as on the one hand^ 

K{x\n) = - log /ip[xf„] +0(1) 

for /ip-random sequences by Levtn-Schnorr theorem, and on the other hand 

log jjplx \n\ = — log(p) — log(l - p) 

n n n 

which converge to h{^p) for ;Up-random sequences, by the Strong Law of Large Numbers 
for Martin-Lof random sequences. 

This result highlights the relationship between Shannon's information theory, Kol- 
mogorov algorithmic information theory and effective randomness. 

Ergodic theory provides a natural extension of information theory in which many 
results can be transferred, with more involved proofs, from the case of independent iden- 
tically distributed random variables to the ergodic case, where independence is only re- 
quired asymptotically, in the average (see Section 2 for a precise definition). 

First, the strong law of large numbers extends to Birkhoff 's ergodic theorem. Second, 
the coincidence between local information and entropy extends through the Shannon- 
McMillan-Breiman theorem. Whether Martin-Lof randomness fits with these theorems 
has been an open problem for a while. The first results were proved by V'yugin [V'y98], 
based on non-classical, constructive proofs of the theorems. He proved, in particular: 

Theorem 1.1 (Effective Birkhoff ergodic theorem I). Let fi be a computable shift-invariant 
ergodic measure over {0, 1}^ and f G L^ijJ.) be computable. For every Martin-Lof ^-random 
sequence x, 

n-l 



lim ^y2f°T\x)= /"/d/i. 

fc=0 "^ 

The entropy of an ergodic measure is defined as 

/i(/i) = lim V fi[w] log fi[w]. (2) 

n— 5>oo fl •'— ' 
\w\=n 

Observe that (1) and (2) are consistent as they give the same quantity when /z is a Bernoulli 
measure. 

Theorem 1.2 (Effective Shannon-McMillan-Breiman theorem I). Let fibea computable shift- 
invariant ergodic measure over {0, 1}^. For every Martin-Lof /j,-random sequence x, 

lim sup — = lim sup log /i[x |~„] = /i(/i) . 

n— ^-oo J^ n— >oo ri 



^K is the prefix version of Kolmogorov complexity 



The question whether liminf ^^ coincides with /i(^) for every Martin-Lof /i-random 
was left open by V'yugin. An alternative proof of Theorem 1.2 approximating ergodic 
measures by Markovian measures was later developed by Nakamura [Nak05], but also 
left the question open. In this paper we provide a positive answer to this question. 

A classical proof of the Shannon-McMillan-Breiman theorem uses Birkhoff 's ergodic 
theorem, applied to some particular functions. The problem in making it effective is that 
these functions are not computable in general. Recent works have been achieved to push 
the effective ergodic theorem to the largest possible class of functions. Here we extend it 
enough to get the full effective Shannon-McMillan-Breiman theorem. 

In Section 2 we recall basic notions of computability, randomness and ergodic theory. 
In Section 3 we develop effective versions of Birkhoff 's ergodic theorem. In Section 4 we 
present our main result. 

2 Background and notations 

We work on the Cantor space {0, 1}^ of infinite binary sequences. A finite word w G 
{0, 1}* determines the cylinder [w] C {0, 1}^ of infinite sequences starting with w. If 
X e {0, 1}^ and n e N, x fn is the prefix of x of length n, and is also denoted XqXi . . . x^-i- 
The cylinders form a base of the product topology. 

Effective topology. An open set U C {0, 1}^ is effective if it is a recursively enumerable 
union of cylinders. A closed set is effective it its complement is an effective open set. 
A function / : {0, 1}^ — )■ M is computable if there is a Turing machine that on oracle x 
and input n computes a rational number q such that \q — f{x)\ < 2~". Equivalently, / is 
computable if for every rational numbers a < b, f~^{a, b) is effectively open, uniformly 
in a, fo. A function / : {0, 1}^ — )■ [0, +oo] is lower (resp. upper) semi-computable if there 
is a Turing machine that on oracle x and input n computes a rational number g„ such 
that f{x) = sup„ Qn (resp. f{x) = inf.„ g„). Equivalently, / is lower (resp. upper) semi- 
computable if for every rational number a, f'^{a, +oo] (resp. /"^[O, a)) is effectively open, 
uniformly in a. 

Kolmogorov complexity and Martin-Lof randomness. For w E {0, 1}*, K{w) is the 
prefix version of Kolmogorov complexity, defined by Levin and Chaitin independently. 
It is defined as the length of a shortest input of a universal Turing machine with prefix-free 
domain computing w on that input. 

A probability measure fi over {0, 1}^ is determined by its value on cylinders fi[w], for 
w E {0, 1}*. i^L is computable if fi[w] is a computable real number, uniformly in w. Given 
a computable probability measure fi, a sequence x E {0, 1}^ is Martin-Lof fi-random, 
denoted x E M L^, if there is c such that for all n, 

K{x\n) > - log fl[x\n] - C. 



Martin-Lof 's original definition [ML66] was expressed in terms of tests rather than com- 
plexity, but the one given here, due to Levin and Chaitin [Cha75] independentely, was 
proved to be equivalent by Levin [Lev73] and Schnorr [Sch73]. 
The function 

t^(x) = SUp{-l0gfl[x\n] - K{x\n)} 

n 

is lower semi-computable and /2*'' d^ < L Moreover, it was proved in [Gac80] that 2*^ 
is maximal in the sense that for every integrable lower semi-computable function / : 
{0,1}^ — )■ [0,+oo], there exists c/ such that / < c/2|j. We call such an / a fx-test. It 
tests Martin-Lof randomness in the sense that x G ML^ iff f{x) < oo for each fx-test f iff 
t^{x) < oo. More can be found on this subject in [Nie09, DHIO]. 

Ergodic theory. We recall some basic notions of ergodic theory, more details can be 
found in [SmoZl, Pet83]. We denote by T : {0, 1}^ -^ {0, 1}^ the shift map defined by 
T{xoXi . . .) = 2^1X2 .... A measure fi over {0, 1}^ is shift-invariant if for all Borel sets A, 
fi{T~^A) = id{A), equivalently if fi[Ow] + I-l[Iw] = fi[w] for all w E {0, 1}*. fi is ergodic if 
for all Borel sets A such that T~'^A = A up to a null sets, fJ-{A) = or 1. Equivalently, /i is 
ergodic if for all u,v e {0, 1}*, 



n— 5>oo n 



_. n— 1 

lim - V /i([u] n T^^H) = fi[u] ■ fi[v]. 



fc=0 



3 Effective ergodic theorems 

The following theorem, taken from [BDMSIO], extends a result of Kucera from the uni- 
form measure to any ergodic shift-invariant measure: 

Theorem 3.1 (Effective Poincare recurrence theorem). Let ft be a computable ergodic shift- 
invariant measure and C C {0, 1}^ an effective closed set such that fJ,{C) > 0. Every Martin-Lof 
fi-random sequence has a tail in C, i.e. for every x e ML^ there exists k such that T^{x) G C. 

In [BDH+10] and [FGMNIO] independently this result was used to prove that not only 
the orbit of x eventually falls into C, but it does so with frequency //(C). 

Theorem 3.2 (Effective Birkhoff ergodic theorem II). Let n be a computable ergodic shift- 
invariant measure and C C {0, 1}^ an effective closed set such that ^(C) > 0. For every Martin- 
Lof jj-random sequence x, 

lim -\{k < n : T^(x) G C}\ = fi{C). 

n—^oo 77, 

We first generalize the result from sets to functions: 

Theorem 3.3 (Effective Birkhoff ergodic theorem III). Let ^ be a computable ergodic shift- 
invariant measure. Assume f : {0, 1}^ — )• [0, +oo] is: 



either lower semi-computable, 

or upper semi-computable and bounded by a ^-test. 



For each x e M L^, 



n-l 



/ t — _L n 

lim V/oT'=(x)= //d/i. 

fc=0 ^ 

Proof. Let us introduce the notation Al{x) = -^{fix) + ... + / o r'^"^(x)). 

If / is lower semi-computable, then there is a sequence of uniformly computable non- 
negative functions /„ /^ /• Applying Theorem 1 . 1 to /„ and x G M L^ gives lim inf ^ A{ (x) > 
lim inf fc v4{" (x) = Jfndfi. By the monotone convergence theorem, Jfn.dfi /^ //d/i, so 
liminffc A'^[x) > Jf d/i. If // d/i = oo we are done. Otherwise, let q > // d/i be a rational 
number. The set Ck := {x : V/c > K, A{(x) < q} is effectively closed and by the classical 
ergodic theorem, there exists K such that ii{Ck) > 0. Theorem 3.1 tells us that if x G ML^ 
then there is n such that T"(x) G Ck- As a result, lim sup A{(x) = limsup A{(r'^(x)) < q. 
As this is true of every q > Jf dfi, we get the result. 

Now, if / is upper semi-computable and f < t where t is a //-test, then for x G M L^, 
applying the preceding result to t and t — f, 

Aiix) = Aiix) - «-^(x)) -> |td/i - jit - /) d/i = y"/d/i. 

D 
We then extend this result further: 

Corollary 3.1 (Effective Birkhoff ergodic theorem IV). Let f : {0, 1}^ -> [0, +oo] be A° on 
ML^, i.e. there is a sequence fn of uniformly computable functions such that f{x) = lim„ /n(x) 
for each x G ML^. Assume that f is dominated by a fi-test. For every x G ML^, 

n-l 



/(. J- n 

lim V/oT'=(x)= //d/i. 

7 n ^ 



n— >oo 

fc=0 



Proof Let ^tat = inf„>Ar/„ and hjy = min(t, sup„>^ /„). On ML^, Qn /" f and /ijv \ /• 
By the monotone and dominated convergence theorem, the convergences hold in L^(/i). 
Applying Theorem 3.2 to g^ and /iat gives the result. More precisely, for every x G M L^ 
and every A^, 

liminf 74{(x) > liminf A^^(x) = gj^dfi 

n n J 

SO 

// d/i = sup gNd-H < lim inf A(^{x) < lim sup A{(x) < inf / /i^r d/i = / / d/x. 
TV J " n ^ i J 



D 



4 The effective Shannon-McMillan-Breiman theorem 

We now present our main result. 

Theorem 4.1 (Effective Shannon-McMillan-Breiman theorem II). Let f.i be a computable shift- 
invariant probability measure. For each x e ML^, 

lim — = lim log/i[xy = h{fi). 

n— 5>oo n n—^oo fl 

A proof of the classical result, stating the result for a.e. x, can be found in [Smo71, 
Pet83]. It makes use of martingale convergence theorems and ergodic theorems. The main 
difficulty in adapting the proof is to make sure that the effective versions of the ergodic 
theorem can be applied. The rest of this section is devoted to the proof of Theorem 4.1. 

An easy calculation shows that 



n-l 







-log IJ.[x\n = 


2_^ fn-l 
k=0 


-koT' 


\x) 


where 














fk{x) := 


-log^[2;o Xi...Xk 


= -log 


/i[xo . . 
Ii[xi . . 


..Xk 
. . Xk 




/o(x) := 


-\ogfi[xo. 








Lemma 4.1. 


fk{x) converge for each x G ML^ 


!• 






Proof. Define the computable martingale 












die) = 2 












d{xo) = —, 


1 

„ 1 







(3) 



forfc> 1, 



,. N ^J'[Xl...Xk\ 

d(xo . . . Xk) = — p 7 for A; > 1. 

ll[Xo ...Xk\ 

By the effective Doob's convergence theorem (see Theorem 7.1.3 on page 270 in [DHIO]), 
for each a: G ML^, d{xo . . . Xk) converges, and so does fk{x) = log d{xQ . . . Xk). □ 

Let f{x) be the limit. We write 

^ -. n— 1 ^ n—1 

— \0gfl[x\n] = - V(/n-l-fc O T'{X) - f O T'{X)) + - V / O T'{x) 

n n ^-^ n ^-^ 

k=0 k=0 

and prove that the first term tends to while the second term converges to // d/i = /i(/i). 
We will use the following lemma (Corollary 2.2 on page 261 in [Pet83], Lemma 4.26 on 
page 26 in [Smo71]). 



Lemma 4.2. /* := sup^ fk G L^. 

As /fc — ;■ / a.e. and the convergence is dominated by /* e L^, /^ — )• / in L^. 
Proposition 4.1. For each x G ML^, 

n-l „ 

lim-V/oT'=(x)= fdfi = h,{P). (4) 

fc=0 "^ 

Proof. That // d/i = /i(^) is a classical result and follows from h{fi) = linifc //fc d/i and the 
L^-convergence of fk to /. 

/* is lower semi-computable and by Lemma 4.2 it is a ^-test. By construction, / is Ag 
on ML^ and it is dominated by /* so it satisfies the conditions of Corollary 3.1, from which 
the result follows directly. D 



Proposition 4.2. For each x G ML^, 

n.-l 
n->-oo n 



n-l 

lim - V /„„i_fc o T\x) - f o T\x) = 0. (5) 

7— inn r7 -i— ' 



fc=0 



Proof. Let 

Forx e ML^, 



qn = sup\fk- f\ and ^tv = sup |/fc - /jl 

k>N k,j>N 



\fk{x)-f{x)\ = lim\fk{x)-f,ix)\ 

= limsup|/fc(x) - fj{x) 
j 

< sup|/fc(x) - fj{x)\, 
j>N 



SO gwix) < gwix). As fk — )■ / a.e., gN ^ a.e. As gN < 2/* G L^ (^at — )■ in L-*^ by the 
dominated convergence theorem. On M L^, 



fc=0 



n ' ^ fi < ^ 

k=0 

^ n-l-N -. n-l 

= - V |/„-i-fcoT^-/oT^| + - V !/„_,_, oT'^-ZoT^ 

k=0 k=n-N 

n-l-N n-l 

n ^ — ^ n ^-^ 

k=0 k=n-N 

^ n-l-N n-l n-N-1 

<- Y. ~g^oT' + -Yir + f)°T'-- Y (r + /)°T^ 



n ^ — ' n ^ — ' n 

fc=0 k=0 k=0 



Fix A^ and let n — t- oo. As ^at G L^ is lower semi-computable, the first term converges to 
J^Tv d/U by the Effective Ergodic Theorem 3.3. As /* + / is A2 on ML^ and is dominated by 
the ^-test 2/*, the second and the third terms converge to /(/* + /) d/i by Corollary 3.1 
so their limits cancel each other. 

As JgN dfj, — > 0, we have proved equality (5). D 

Putting equalities (3), (4) and (5) together gives, for x E M L^, 

1 1 "'^ 

lim log/x[a:f„] = lim - V" /„_i_fc o r''(x) = h{^). 



n n n 

k=0 
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